System and method for augmentation of surgery

ABSTRACT

The system and method includes a manipulator for manipulating a surgical instrument relative to a patient&#39;s body and, a position sensor for sensing the position of the surgical instrument relative to the patient&#39;s body. The manipulator can be manually or computer actuated and can have brakes to limit movement. In a preferred embodiment, orthogonal only motion between members of the manipulator is provided. The position sensor includes beacons connected to the patient and manipulator or surgical instrument and, a three dimensional beacon sensor adapted to sense the location and position of the beacons. Redundant joint sensors on the manipulator may also be provided. The system and method uses a computer to actively interact with the surgeon and can use various different input and output devices and modes.

This is a divisional of application No. 08/147,008, filed Nov. 2, 1993,which is a continuation of application No. 07/714,816, filed Jun. 13,1991; abandoned.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to surgery and more particularly, to asystem and method for positioning, moving and locating surgicalinstruments for performing surgery on a patient.

2. Prior Art

Recent advances in medical imagining technology (CT, MRI, PET, etc.),coupled with advances in computer-based image processing and modellingcapabilities have given physicians an unprecedented ability to visualizeanatomical structures in live patients, and to use this information indiagnosis and treatment planning. The precision of image-basedpre-surgical planning often greatly exceeds the precision of actualsurgical execution. Precise surgical execution has been limited toprocedures, such as brain biopsies, in which a suitable sterotacticframe is available. The inconvenience and restricted applicability ofsuch a frame or device has led many researchers to explore the use ofrobotic devices to augment a surgeon's ability to perform geometricallyprecise tasks planned from computed tomography (CT) or other image data.The ultimate goal of this research is partnership between a man (thesurgeon) and machines (computers and robots), that seeks to exploit thecapabilities of both, to do a task better than either can do alone.Machines are very precise and untiring and can be equipped with anynumber of sensory feedback devices. Numerically controlled robots canmove a surgical instrument through an exactly defined trajectory withprecisely controlled forces. On the other hand, the surgeon is verydexterous. He is also quite strong, fast, and is highly trained toexploit a variety of tactile, visual, and other cues. "Judgementally"controlled, the surgeon understands what is going on in the surgery anduses his dexterity, senses, and experience to execute the procedure.However, the surgeon usually wants to be in control of everything thatgoes on. If the surgeon desires to increase precision within acceptablelimits of time or with sufficient speed, he must be willing to rely onmachines to provide the precision.

One potential problem with a robotic device is undesired motion. Themost obvious way to prevent a robotic device from making an undesiredmotion is to make it incapable of moving of its own accord. Motor-lessmanipulators have been implemented in the past which use joint encodersto provide feedback to the surgeon on where his instruments are relativeto his image-based surgical plan. European Patent Application 326,768A2describes one such device. One important limitation of this approach isthat it is often very difficult for a person to align a tool accuratelyin six degrees-of- freedom with only the use of positional feedback.Passive manipulators, permitting free motion until locked, have alsobeen implemented in the past for limb positioning, tissue retraction,instrument holding, and other applications in which accuracy is notimportant. A three degree-of-freedom passive manipulation aid forprostate surgery has also been used clinically in the past.

In cases where only a single motion axis is required during the "incontact" phase of the surgery, a robot has been used in the pastessentially as a motorized sterotactic frame. A passive tool guide isplaced at the desired position and orientation relative to the patient.Brakes are applied and robot power is turned off before any instrumenttouches the patient. The surgeon provides whatever motive force isneeded for the surgical instruments themselves and relies on his owntactile senses for further feedback in performing the operation. Thisapproach ameliorates, but does not entirely eliminate, the safety issueraised by the presence of an actively powered robot in close proximityto to the patient and operating room personnel. Furthermore, maintainingaccurate positioning is not always easy, since many robots tend to "sag"a bit when they are turned off or to "jump" when brakes are applied.Leaving power turned on and relying on the robot's servocontroller tomaintain position introduces further safety exposures. Finally, thistype of approach is limited to cases where a fixed passive guidesuffices. The surgeon cannot execute a complex pre-computed trajectoryby use of this approach, nor can he precisely relocate an instrument orbody part from one place to another.

Over the past several years, researchers at IBM and the University ofCalifornia at Davis developed an image-directed robotic system toaugment the performance of human surgeons in precise bone machiningprocedures in orthopedic surgery, with cementless total hip replacementsurgery as an initial application. This application inherently requirescomputer controlled motion of the robot's end-effector while it is incontact with the patient. Thus, considerable attention had to be paid tosafety checking mechanisms. In-vitro experiments conducted with thissystem demonstrated an order of-magnitude improvement in implant fit andplacement accuracy, compared to standard manual preparation techniques.A clinical trial on dogs needing hip replacement operations is presentlyunderway.

It is the objective of the present invention to provide a new andimproved system and method for augmentation of surgery.

SUMMARY OF THE INVENTION

The foregoing problems are overcome and other advantages are provided bya new and improved system and method for augmentation of surgery.

In accordance with one embodiment of the present invention, an apparatusfor use in moving an end effector is provided. The apparatus comprises afirst manipulator and a second manipulator. The first manipulatorcomprises a coarse motion manipulator and a fine motion manipulator. Thesecond manipulator is connected to a distal end of the firstmanipulator. The second manipulator has means for providing orthogonallydecoupled degrees of freedom with a common remote center-of-motionlocated at a work point some distance from the manipulator mechanism,means for selectively locking or releasing the separate degrees offreedom, and arranged so that small rotational realignments of an endeffector connected to the end of the second manipulator can be providedwithout requiring large motions of any manipulator joint. This mechanismprovides at least three orthogonally decoupled revolute degrees offreedom, together with additional linear degrees of freedom.

In accordance with another embodiment of the present invention, a systemfor manipulating movement of a surgical instrument is provided. Thesystem comprises a mechanical positioner, a computer controlled brake, acomputer, and means for signaling the computer. The mechanicalpositioner is adapted to have the surgical instrument connected theretoand comprises a plurality of members connected to each other in a serieswith at least one motion joint between two of the members. The computercontrolled brake is located at the motion joint. The computer isconnected to the brake for selectively actuating the brake upon anoccurrence of a predetermined event. The means for signaling thecomputer can signal the computer of the occurrence of the predeterminedevent.

In accordance with another embodiment of the present invention, a systemfor assisting the surgeon in positioning a surgical instrument relativeto a target position is provided. The system comprises means formanipulating the position of a surgical instrument, means for sensingthe position of the surgical instrument, means for determining a pathfrom a sensed position of the surgical instrument to the targetposition, and means for audibly signaling deviation of the position ofthe surgical instrument from the path.

In accordance with another embodiment of the present invention, a systemfor assisting a surgeon in positioning an article relative to a targetposition is provided. The system comprising means for determining asurgical plan based upon input patient information, means for sensingsurgical execution of the surgical plan by the surgeon, means foradvising the surgeon and means for automatically selecting differenttypes of advice.

The means for advising the surgeon can advise the surgeon based uponcomparison of the surgical plan and the sensed surgical execution. Themeans for automatically selecting different types of advice can selectdifferent types of advice to give the surgeon based upon the surgicalplan and the sensed surgical execution.

In accordance with another embodiment of the present invention, a systemfor assisting a surgeon during surgery is provided. The system comprisesmeans for determining a surgical plan based upon input patientinformation, means for sensing surgical execution of the surgical planby the surgeon, means for advising the surgeon, and means for inputtinga change in the surgical plan. The means for advising the surgeon canadvise the surgeon of the surgical plan and the sensed surgicalexecution during the surgery. The means for advising comprises acomputer. The means for inputting a change in the surgical plan caninput a change in the surgical plan into the computer during surgery anddetermine a new surgical plan based, at least partially, upon previouslysensed surgical execution of the surgical plan.

In accordance with another embodiment of the present invention, anapparatus for moving a surgical instrument relative to a patient isprovided. The apparatus comprises a base, a first link, a second link, athird link, a fine adjustment manipulator, and means for selectivelylocking and unlocking movement. The first link is movably mounted to thebase for movement along a first axis of motion. The second link ismovably mounted to the first link for movement along a second axis ofmotion perpendicular to the first axis of motion. The third link ismovably mounted to the second link for movement along a third axis ofmotion perpendicular to the first and second axes of motion. The fineadjustment manipulator comprises a first section movably mounted to thethird link along a fourth axis of motion. The fine adjustmentmanipulator also comprises a second section movably mounted to the firstsection along a fifth axis of motion perpendicular to the fourth axes ofmotion, and a third section movably mounted to the second section alonga sixth axis of motion perpendicular to the fourth and fifth axes ofmotion. The means for selectively locking and unlocking can lock andunlock movement of individual links and sections to provide orthogonalonly movement between connected links and sections upon unlocking of thelinks and sections.

In accordance with another embodiment of the present invention, asignaling device for use with a position monitoring system for use in asurgical operation is provided. The device comprises a pin, and abeacon. The pin has a center axis and is adapted to be positioned into apatient's body. The beacon has a housing and an energy emitter. Thehousing has a channel adapted to receive an end of the pin therein, achannel axis coaxial with the energy emitter, and means for positioningthe housing at a predetermined position on the pin such that the energyemitter is coaxially aligned with the center axis of the pin whenconnected thereto.

In accordance with another embodiment of the present invention, a systemfor assisting a surgeon during surgery is provided. The system comprisesmeans for sensing a surgical operation, and means for automaticallyselecting different sensing modes. The means for sensing includesdifferent modes of sensing. The means for automatically selectingdifferent sensing modes can select the different sensing modes basedupon the sensed surgical operation or on the sensed progress of thesurgical operation relative to a predefined surgical plan.

In accordance with one method of the present invention, a method ofassisting a surgeon in positioning a surgical instrument relative to apatient's body during surgery comprises steps of connecting beacons tothe patient; providing a passive manipulator for moving the surgicalinstrument; connecting the surgical instrument to the manipulator;connecting beacons to the manipulator; determining the positions of thebeacons on the patient relative to a portion of the patient's body; andsensing movement of the beacons on the manipulator relative to thebeacons on the patient and thus, sensing the movement of the manipulatorrelative to the portion of the patient's body.

In accordance with another method of the present invention a method ofsensing relative positioning of multiple portions of a patient's bodyduring surgery is provided. The method comprises steps of connectingbeacons to portions of the patient's body; measuring the locations ofthe beacons; sensing the position of reference points on the patient'sbody; determining the position of the beacons relative to the sensedreference points; tracking the movement of the beacons; and determiningthe positions of the portions of the patient's body relative to eachother based upon the relative movement of the beacons to each other.

In accordance with another embodiment of the present invention, anapparatus for assisting a surgeon in manipulating a surgical instrumentintended to be placed into a patient's body through an opening in whichthere is a relatively small amount of lateral motion at the opening isprovided. The apparatus comprises a first manipulator; and a secondmanipulator. The second manipulator is connected to the firstmanipulator and comprises at least three orthogonally decoupled revolutedegrees of freedom and is positionable relative to the opening toprovide a center of motion at the opening, and means for connecting thesurgical instrument to the second manipulator. The means for connectingmay include a sliding link for axially sliding the surgical instrumentto provide an apparatus that provides a spherical work volume centeredat the opening.

In accordance with another method of the present invention, a method oftracking position of a surgical instrument inside a patient's body isprovided comprising steps of inserting a first surgical instrument intoa patient's body, the first instrument having a visual target thereon;inserting a second surgical instrument into the patient's body, thesecond instrument having means for transmitting an image out of thepatient's body; and moving at least one of the surgical instrumentsautomatically based upon the movement of the other surgical instrumentsuch that the visual target can be substantially constantly transmittedby the means for transmitting an image.

DETAILED DESCRIPTION OF THE DRAWINGS

The foregoing aspects and other features of the invention are explainedin the following description, taken in connection with the accompanyingdrawings, wherein:

FIG. 1A is a schematic perspective view of a positioning apparatusincorporating features of the present invention.

FIG. 1B is a schematic view of a lock or brake with a micrometeradjuster.

FIG. 2 is a schematic view of a system incorporating features of thepresent invention having the positioning apparatus shown in FIG. 1.

FIG. 3 is a schematic view of an alternate embodiment of the presentinvention.

FIG. 4 is a schematic view of an alternate embodiment of the fine motionmanipulator of the embodiment shown in FIG. 1 or FIG. 3.

FIG. 5 is a perspective view of a beacon and fixation pin incorporatingfeatures of the present invention.

FIG. 6 is a schematic view of beacons for use with the present inventionattached to a portion of a patient's anatomy.

FIG. 7 is a schematic perspective view of a pointer for use in a systemincorporating the present invention.

FIG. 8 is a schematic view of an alternate method for using features ofthe present invention.

FIG. 9 is a schematic view of a fine motion rotational manipulator withan instrument inserted through an incision into a patient.

FIG. 10 is a schematic view of a system for use in laparoscopy orsimilar procedures.

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1A, there is shown a schematic perspective view of apositioning apparatus 10 incorporating features of the presentinvention. Although the present invention will be described withreference to the embodiments shown in the drawings, it should beunderstood that the present invention is capable of being used invarious different types of embodiments and methods. In addition, anysuitable size, shape, or type of elements or materials can be used. Asused herein, the terms "position" or "positioning" is intended to meanlocation and/or orientation. Also as used herein, the term "orthogonallydecoupled" is intended to mean that small individual motions ofmanipulator joints affect only single, orthogonal degrees of freedom ofthe manipulator's end effector or of an object held in the manipulator'send effector. Thus, individual joints may be used easily, intuitively,and independently to align degrees of freedom of the end-effector toachieve a desired goal. Two linear degrees of freedom are orthogonal ifthey are perpendicular. Similarly, two revolute degrees of freedom areorthogonal if the directions of their axes of rotation areperpendicular. Further, a revolute degree of freedom about a line axisis orthogonally decoupled from linear degrees of freedom resolved at apoint M if the line axis "a" passes through the point M. The mechanismsdescribed herein provide orthogonally decoupled motion resolved at apoint located some distance from the mechanical structure. Typically,they comprise, serially, a mechanism providing three orthogonallydecoupled linear motions, followed by a remote center-of-motion "wrist"providing orthogonally decoupled revolute motions, followed by anadjustable tool holding mechanism permitting a desired tool or body partto be placed at a desired position and orientation relative to the wristcenter of motion. Typically, the successive axes-of-rotation of thewrist are perpendicular and intersect at the center of motion, so thatany rotation about any wrist axis introduces no translationaldisplacement of a tool or body part held at the center of rotation.Furthermore, the wrist is arranged so that any desired smallreorientation of the tool or body part held at the center-of-rotationmay be accomplished with only small angular motions about the individualwrist axes. In many cases, the mechanisms described herein comprisekinematically redundant instances of linear and rotationalsub-mechanisms to support a variety of coarse-fine manipulation methods.

The positioning apparatus 10, in the embodiment shown, generallycomprises a bulk or coarse motion manipulator 12 and a fine motionmanipulator 14. The apparatus 10 is generally intended to be used in anoperating room. A similar system is disclosed in copending U.S. patentapplication Ser. No. 07/523,611 filed May 11, 1990 entitled"Image-Directed Robotic System For Precise Robotic Surgery IncludingRedundant Consistency Checking", now U.S. Pat. No. 5,086,401, assignedto the same assignee as herein which is hereby incorporated by referencein its entirety herein. The coarse motion manipulator 12 generallycomprises three links or sections; first link 16, second link 18, andthird link 20. In the embodiment shown, the first link 16 is movablymounted on a base having a rail system 22. In a preferred embodiment,the base 22 would be stationarily positioned on a floor of an operatingroom to stationarily position the positioning apparatus 10 during asurgical operation. Alternatively, the base 22 could be fixed to anoperating table. In the embodiment shown, the base 22 comprises tworails 24 and two end blocks 26 at opposite ends of the rails 24. Thefirst link 16 has a main block 27 with rail channels 28 that the rails24 pass through such that the first link 16 can ride along the base 22along a linear path or axis as shown by arrow A. A suitable manualposition lock (not shown) is provided on the block 27 to selectivelystationarily position or lock the first link at a selected position onthe base 22. The end blocks 26 establish a limited range of motion forthe first link 16 on the base. The first link 16 also comprises a secondrail system 30 stationarily connected to the block 27. The second railsystem 30 generally comprises two rails 32, two end blocks 34, and beam35 that connects the rail system to the block 27. As can be seen in FIG.1A, the second rail system 30 is generally perpendicular to the firstrail system 22 and moves with the block 27 as it moves along the firstrail system 22.

Although the apparatus 10 shown in FIG. 1A is being described as amanual passive movement apparatus, it should be understood that anymanual locks described could be replaced by a computer controlled brake.This computer controlled brake can be either a simple binary lock or mayprovide a variable, controlled braking force. Furthermore, it should bemade clear that any lock controlling a single degree-of-freedom could beaugmented by a suitable lock with auxiliary micrometer adjustmentpermitting fine positional adjustment after the lock is applied. Itshould be understood that "micrometer adjustment" may comprise anysuitable type of very precise fine motion adjustment device, and is notlimited to a fine pitch lead screw, such as is commonly found oncommercial micrometer stages. Micrometer adjustment can be provided atany joint, either in the coarse or fine manipulators, and may be eithermanually controlled or computer controlled, or both. Finally, it shouldbe made clear that such micrometer adjustments can be manually orcomputer activated and controlled. One typical mechanism is illustratedschematically in FIG. 1b, although other implementations are possible.In this schematic, moving members 801 and 802 are constrained byappropriate bearing structures so that they move in only a single degreeof freedom X relative to fixed member 803. Locking mechanism 804 locksmoving member 801 to fixed member 803. Micrometer lead screw 805provides a means of adjusting the relative displacement alongdegree-of-freedom X of moving member 802 relative to moving member 801.Thus, when moving member 801 is locked relative to fixed member 803,micrometer lead screw 805 can be used to adjust the position of movingmember 802 relative to fixed member 803.

The second link 18, in the embodiment shown, generally comprises acolumn shaped frame 36 with a bottom end movement block 38 and a top endthird rail system 40. The bottom end movement block 38 has rail channels42 through which the second rail system rails 32 project through. Thissystem allows the bottom end movement block 38, and thus the second link18, to move along the second rail system 30 in the linear path or axisas shown by arrow B. As can be seen, the linear path of the second link18 is orthogonal to the linear path of the first link 16 as shown witharrow A. The end blocks 34 allow the second link 18 to have a limitedrange of movement on the second rail system 30. The bottom end movementblock 38 also comprises a manually actuatable lock 44 to enable the userto selectively lock the relative position of the second link 18 to thefirst link 16. The top end third rail system 40 is similar to the firstand second rail systems and generally comprises two rails 46 and two endblocks 48 which are stationarily connected to the second link frame 36.The second link 18, in the embodiment shown, also includes a counterbalance system 50 which will be further described below.

The third link 20, in the embodiment shown, generally comprises a frame52, a movement block 54 at a first end proximate the second link 18, anda distal end 56. The movement block 54, in the embodiment shown, hasrail channels 58 which the rails 46 pass through. The movement block 54is slidably positioned on the rails 46 for movement in the verticaldirection as shown by arrow C with the end blocks 48 providing a limitto the range of motion of the third link 20. Due to the forces ofgravity, the counterbalance system 50 is provided to counter balance theeffects of gravity on the third link 20, thus making the third link 20substantially easier for a surgeon to move. The counterbalance system50, in the embodiment shown, generally comprises dowels 60 stationarilyconnected to the frame 36 of the second link and constant force springs62. First ends 64 of the springs 62 are fixedly connected to themovement block 54. The remainder of the springs 62 are coiled around thedowels 60 and have an undeformed coil shape. As the third link 20 ismoved from the top of the third rail system 40 towards the bottom of thethird rail system. The movement block 54 pulls on the first ends 64 ofthe springs 62 and pulls the springs 62 out from the dowels 60 causingthe springs 62 to deform into a relatively straight shape. Theresistance of the springs 62 to deformation counterbalances the weightof the third arm 20 and the apparatus attached to its distal end 56. Themovement block 54 also comprises a manually actuatable lock 66 which canbe used by the surgeon to selectively lock the position of the thirdlink 20 relative to the second link 18.

The coarse motion manipulator 12 as described above is generallyintended to be used as a passive manipulator for moving the fine motionmanipulator 14 and surgical instrument connected thereto. Thus, thesurgeon can move each one of the links, either successively or together,to roughly or generally position a surgical instrument. The locks at thejoints between the links can be used to prevent relative motion betweenthe links. These locks can be actuated either successively or together.It should be understood that any suitable type of relative motion systemcould be used between the links; not merely rail systems. The linkscould also be pivotally connected to each other. However, in a preferredembodiment the three links are capable of orthogonal only motionrelative to each other. This orthogonal only motion ordegrees-of-freedom assists the surgeon in positioning or moving thecoarse motion manipulator 12 due to the restricted path of motionavailable to each link. Thus, the surgeon can easily foresee the effecton the fine motion manipulator 14 if one of the links of the coarsemotion manipulator is moved. In a system where computer controlled locksor brakes are provided with the apparatus 10, one embodiment of such acomputer-controlled position lock would be an electric particle brakeoperating through a rack-and-pinion transmission mechanism. Such anembodiment permits the computer to control the braking force from zeroor very low drag up to full locking. Such variable braking force can beused both to provide additional tactile feedback to the surgeon and toprovide additional damping to assist him in positioning the mechanism.If desired, additional manually actuated position locks may be providedwith the computer-actuated position locks.

The fine motion manipulator 14, in the embodiment shown, generallycomprises a linear motion section 68 and a rotational manipulatorsection 70. The linear motion section 68, in the embodiment shown,generally comprises three sections; a first section 72, a second section74, and a third section 76. A mounting block 78 is provided at thedistal end 56 of the third link 20 for mounting the fine motionmanipulator 14 to the coarse motion manipulator 12. In the embodimentshown, stationarily connected to the mounting block 78 is a race member80. Movably mounted to the race member 80 is the first section 72. Thefirst section 72 can move relative to the race member 80 in a linearpath or axis as shown by arrow B'. This linear path B' is parallel tothe path B of the second arm 18, but substantially smaller in allowabletravel path length. The actual mechanical connection between the firstsection 72 and race member 80 is provided by the top of the firstsection 72 having an interlocking engagement with the race member 80 toprevent movement in all directions except along path B'. Suitable rollerbearings are provided in the joint between first section. 72 and racemember 80 to provide relatively smooth motion. In the embodiment shown,the connection of the first section 72 to the race member 80 alsocomprises a lock (not shown) which can be selectively manually actuatedby a surgeon to lock or fix the relative positions thereof.

The bottom of the first section 72 has the top of the second section 74movably connected thereto such that the second section 74 can moverelative to the first section 72. The bottom of the first section 72 hasa linear raceway. The top of the second section 74 also has a linearraceway that interlocks with the first section bottom raceway to preventmovement in all directions except along path A'. Suitable rollerbearings or the like and stops or ledges are provided between the firstand second sections to provide a smooth movement. A lock 82 is providedat the joint between the first and second sections such that a surgeoncan selectively manually lock the two sections to each other. Themovable path of the second section 74 as shown by arrow A' is orthogonalto the movement path of the first section 72 as shown by arrow B' and,is parallel to but substantially smaller than the path length of thefirst arm 16 shown by arrow A. The bottom of the second section 74 has aframe structure 84 fixedly connected thereto. A side of the framestructure 84 has the third section 76 movably mounted thereon formovement along a substantially vertical path or axis as shown by arrowC'. The path of the third section 76 is parallel to, but substantiallyless than the path C of the third arm 20 on the second arm 18. The sideof the frame structure 84 has a linear raceway thereon that interlockswith a linear raceway on the third section 76 to prevent movement in alldirections except along path C'. A zero-force spring counterbalancemechanism (not shown) similar to counterbalance mechanism 50 is providedat the third section 76 which compensates for the weight of the entiremechanism distal to frame structure 84. Suitable rollers are providedbetween third section 76 and frame structure 84 for a smooth motion. Inthe embodiment shown, a manual lock 86 is provided at the joint of thethird section 76 and frame structure 84 to control relative movementtherebetween. Additionally, a manually actuatable micrometer adjustercould be provided at the joint. The first and second sections 72 and 74could also include micrometers at their joints. The third section 76, inthe embodiment shown, has a general "L" shape with a bottom having therotational manipulator section 70 connected thereto. Thus, when any ofthe sections 72, 74 or 76 are moved, the rotational manipulator section70 is moved.

One of the features of the present invention is the orthogonal onlyredundant linear motion system provided by the coarse motion manipulator12 and fine linear motion section 68. The orthogonal only motion of thesections 72, 74 and 76, similar to the orthogonal only motion of thecoarse motion manipulator 12, assists the surgeon in manipulation of thefine linear motion section 68 due to the restricted path of motionavailable to each section. Thus, the surgeon can easily foresee theeffect if one of the sections is proposed to be moved and, control ofmovement is simplified. Although redundant linear motion is describedabove, it should be understood that this need not be provided. Arotational coarse motion manipulator may be provided between manipulator12 and manipulator 14. This, of course, would be more useful to alignthe fine motion axes so that they are conveniently located for aparticular manipulation task.

Also similar to the coarse motion manipulator 12, the fine linear motionsection 68 is generally intended to be used as a passive manipulator formoving the rotational manipulator section 70. The surgeon can move eachone of the sections either successively or together to finely positionthe section 70 after having been moved by the coarse motion manipulator12. This redundant motion system can allow a surgeon to move a surgicalinstrument into position much faster than a single motion system andwith good accuracy due to fast motion provided by the coarse motionmanipulator 12 and the accurate motion provided by the fine linearmotion section 68. The accuracy provided by the fine linear motionsection 68 allows the coarse motion manipulator 12 to be moved fasterbecause of the decrease in accuracy needed to position the coarse motionmanipulator 12. Another advantage of the coarse/fine structure is thatit permits relatively large work volumes while limiting the inertia thatthe surgeon must cope with. The modularity is similarly very useful forexperimentation.

The rotational manipulator section 70, in the embodiment shown,generally comprises a roll section 88, a pitch section 90, a yaw section92, and an end section 94. The roll section 88 generally includes tworace plates 96 and 98 that are movably connected to each other toprovide rotation about the vertical axis as shown by arrow E. The topplate 96 is fixedly connected to the third section 76. A suitable manualor computer controllable lock is provided to fix or lock the two plates96 and 98 together. In the embodiment shown, the locking mechanismcontains a micrometer adjustment mechanism 100 to provide extremely fineadjustments of the angle between plates 96 and 98 after the lock isapplied. The pitch section 90 is a goniometer cradle which has twocurved plates 102 and 104 movably connected to each other. The top plate102 is fixedly connected to the roll section bottom plate 98. The bottomplate 104 is adapted to move along a curved or revolute path relative tothe top plate 102 as shown by arrow F. The yaw section 92 is agoniometer cradle which includes two curved plates 106 and 108 that aremovably connected to each other. The top plate 106 is fixedly connectedto the pitch section bottom plate 104. The bottom plate 108 is adaptedto move in a curved or revolute path relative to the top plate 106 asshown by arrow G. The individual rotational manipulator sections, in theembodiment shown, are each limited to motion in their receptive relativepaths of motion E, F and G. The axes of rotation of the paths E and Fare perpendicular, as are the axes of rotation of the paths F and G.

Furthermore, the axis of path E is perpendicular to axis of path G whenpitch section 92 is in its normal center of motion. All three axesa_(E), a_(F), and a_(G) of paths E, F, and G intersect at a motioncenter point M at a convenient distance from the fine manipulatormechanism. Thus, the rotational manipulator section 70 providesorthogonally decoupled rotational motions resolved at this motion centerpoint. Further, the rotational motions resolved at this point areorthogonally decoupled from the translational motions of the moreproximal translational sections (i.e., coarse motion section 12 and finelinear motion section 68). Suitable manual actuated locks are providedat each section to provide selective movement. Alternatively oradditionally, micrometers can be provided at each section. In addition,single additional redundant sections could be provided.

As noted above, suitable locks are provided at each section to provideselective motion. It should also be understood that automatic computeractuated locks and/or manual locks could be provided in both the coarseand fine motion manipulators. One embodiment of such acomputer-controlled position lock could be an electric particle brakeoperating through a rack-and-pinion transmission mechanism. Such anembodiment would permit the computer to control the braking force fromzero or very low drag up to full locking. Such variable braking forcecan be used both to provide additional tactile feedback to the surgeonand to provide additional damping to assist him in positioning themechanism. If desired, both manually actuated and computer actuatedposition locks or brakes may be provided.

It will thus be seen that the embodiment shown supports a coarse-finemanipulation strategy. In this strategy, the coarse manipulator 12 ispositioned so that the remote center-of-motion of the fine manipulator14 is located at a desired position and the motion axes A' and B' (orA', B', and C' if an alternative embodiment such as a ball-and-socketjoint is substituted for revolute joint between manipulators 12 and 14are aligned conveniently for the surgeon). The coarse manipulator 12 isthen locked, and the fine manipulator 14 is adjusted so that axes a_(E),a_(F), and a_(G) of the revolute paths E, F, and G are alignedconveniently and are mutually orthogonal. Finally, the joints at therevolute paths E, F, and G and the linear paths A', B', and C' are usedto perform the desired fine positioning or realignment motions. Thesejointts may be selectively locked or freed as necessary to assist incarrying out the desired motions. Because the fine motions areorthogonally decoupled, readjustment of one degree-of-freedom will notaffect the alignment of other degrees-of-freedom, thus significantlyspeeding up the process of achieving a desired alignment. Furthermore,the process is relatively insensitive to small errors in the placementof the coarse manipulator, unless large rotational adjustments are to bemade. For example, suppose there is an error of 0.5 mm in the placementof the coarse manipulator relative to a desired center of motion. Then,a rotational realignment of 15° will produce only about 0.13 mmundesired motion of the rotation center, which is often negligible orwhich may easily be compensated by fine adjustments of linear motionaxes A', B', and C'. A related advantage of the bulk/fine structure isthat is permits relatively large work volumes while limiting the inertiathat the surgeon must cope with. The modularity is similarly very usefulfor experimentation.

In the embodiment shown, the end section 94 is fixedly connected to thebottom plate 108 of the yaw section 92 and includes means for mountingthe extension tool 110 thereon and, beacons 112. The extension tool 110is known in the surgical art. The tool 110 has two arms 114 and 115 witha center clamp control 116 that is adapted to fix the two arms relativeto each other and, clamps at the ends of the arms 114 and 115 alsocontrolled by the control 116. The clamp at the end of the first arm114, in the embodiment shown, fixedly clamps onto the end member 94. Theclamp 117 at the second arm 115 is adapted to clamp onto a surgicalinstrument (not shown). In normal use, the tool 110 can be adjusted suchthat a desired point on the surgical instrument or a desired point on abody part held on the surgical instrument is located at the center ofmotion M of the manipulator 70. However, the tool 110 need not be used.A surgical instrument may be directly connected to the end section 94or, any suitable intermediate mechanism may be provided. The beacons112, in the embodiment shown, are generally comprised of light emittingdiodes. However, any suitable type of energy emitters or reflectors maybe used. The diodes 112 are used to signal the position, bothorientation and location, of the end section 94.

Suitable calibration procedures may be used to determine the position ofthe remote center-of-motion of the revolute manipulator section 70relative to the beacons. In one such procedure, all joints proximal tothe manipulator section 70 are locked. The joints of the revolutemanipulator section 70 are then all moved to multiple positions and thepositions of the beacons are measured by sensor 122 (described below).Each beacon will move about on the surface of a sphere whose centercoincides with the remote center-of-motion of the revolute manipulatorsection 70, although the radius of each beacon's sphere-of-motion maydiffer from that of another beacon's sphere-of-motion. Standardmathematical techniques may be used to compute this common center fromthe measured beacon positions.

Referring also to FIG. 2, there is shown one embodiment of a system 120incorporating the apparatus 10 shown in FIG. 1. The system 120 generallycomprises the apparatus 10, a sensor 122, a computer 124, an audiosystem 126, and a visual display 128. The beacons 112 (see FIG. 1) onthe end section 94 are connected to the computer 124 by electrical cable130. The computer 124 sends signals to the beacons 112 such that theyare activated in predetermined patterns of time and frequency. In analternate embodiment different beacon colors could be used todistinguish the beacons from each other, thus permitting all beaconpositions to be measured simultaneously. The sensor 122, in theembodiment shown, is an optical sensor that can digitize opticalinformation and transmit the information via cable 132 to the computer124. In a preferred embodiment, the sensor 122 is an OPTOTRAK 3Ddigitizer. OPTOTRAK is a trademark of Northern Digital Corporation. Thesensor 122 uses three CCD linescan cameras 134 to track the active LEDbeacons. This system is fast, accurate, much less readily confused bystray light than similar lateral-cell based devices, and unlikeelectromagnetic field 6D sensors, is unaffected by metal in theoperating theatre. The sensor 122 has been found capable of producing1000 3D positions per second to an accuracy of about ±0.1 mm and ofreturning up to eight 6D positions with an additional time of about 10ms beyond the 3D sampling time. The computer 124, in the embodimentshown, can calculate the positions of the end section 94 based upon thelight from the beacons 112 sensed at the sensor 122. The audio system126 is connected to the computer 124 and has a speaker 136 and an input138 for use with a voice recognition system in the computer. The speaker136 may be used to output computer-generated synthetic speech,prerecorded voice messages, or simple tonal or other audio cues.However, the audio system 126 need not be provided. The visual display128 is connected to the computer 124 and is used to display informationto the surgeons. The display 128 can be located in any suitable locationin the operation room.

The system 120, in the embodiment shown, is generally capable of sensingthe position, both location and orientation, of the end section 94. Inone embodiment, the position of the surgical instrument relative to theend section 94 is input into the computer 124 prior to use of thesurgical instrument. The computer 124 is thus capable of determining orcalculating the position, both location and orientation, of the surgicalinstrument when it is moved. In an alternate embodiment, beacons arealternatively or additionally connected directly to the surgicalinstrument. Thus, in this alternative embodiment, the sensor 122 iscapable of directly sensing the position of the surgical instrument whenit is moved. The system can also be used to keep track of all of thesurgical instruments used in an operation to insure that all instrumentsare properly recovered. The system can obviously be used for a varietyof purposes, but one of the significant advantages of the system is as ameans for determining the relative position of a surgical instrumentrelative to the patient and, as a means for determining the relativeposition of a portion of the patient's anatomy relative to anotherportion of the patient's anatomy or a target area. For this purpose,beacons can be directly connected to the patient or portions of thepatient's anatomy as further described below. Unlike the mechanicalcomputer-aided surgery apparatus in the prior art, such as disclosed inEuropean Patent Application 326768A2, the optical system described aboveprovides a much less cumbersome system adapted for use in a multitude ofdifferent applications.

A preferred system architecture for the present invention can begenerally described as two subsystems or groups; a presurgical modelingand planning system and method, and the surgical system substantially asdescribed above and below. The presurgical system and method generallyuses models of patient information to assist the surgeon in planningprecise surgical procedures. Any suitable type of presurgical system andmethod could be used. In one type of presurgical procedure, describedbelow for informational purposes only, the principal components of thispresurgical procedure include a medical image database and displaysystem, an anatomical model builder, an anatomical feature extractor, asurgical simulator, an anatomical data base, and a surgical planoptimizer. These components are used to produce a surgical plan. Themedical image database and display system supports archival, retrieval,low-level processing, and display of CT, MRI, and other images. Theanatomical model builder transforms recorded images into 3D solid modelsof the patient's anatomy. The process proceeds in three steps. In thefirst step (segmentation), each voxel in the CT data set is assigned atissue classification label, based on an adaptive thresholdingtechnique. In the second step (model reconstruction), a winged-edgeboundary representation of each connected set of tissue classified as"bone" is constructed by a variation of Baker's "weaving wall"algorithm. In the third step (model simplification), coplanar faces aremerged to reduce the size of the model somewhat. The anatomical featureextractor identifies anatomical features from the models. These featuresinclude standard morphometric landmarks, ridge curves, and surfacepatches bounded by ridge curves and geodesics between landmarks. Thepresent implementation is semi-automatic. A technician "seeds" thesearch by identifying points on or near ridge curves, and the computerthen locates and follows the ridge curves. A more automatic proceduremay of course be used. The surgical simulator permits a surgeon tointeractively specify where he wishes to cut the bones apart and tomanipulate the pieces graphically. It also permits him to display thebone fragment motions computed by the plan optimizer (described below)and to modify the plan as he chooses. The anatomical data basesummarizes anatomical feature information for "normal" individuals. Thesurgical plan optimizer uses information from the anatomical data baseto compute optimal motions of each bone fragment to most closelyapproximate the corresponding anatomy of a "normal" individual of thesame age, race, sex, and size as the patient. The surgical plan producedconsists of the model data, the location and sequence of the cuts, thelocation of key anatomical features to be used in registering thepatient to the model data and the planned optimal motion of each bonefragment.

The surgical system and method preferably includes a surgeon interfacesystem and passive manipulation aids. The surgeon interface uses avariety of modalities such as graphics, synthesized voice, tonal cues,programmable impedance of manipulator joints, etc., to provide online,realtime "advice" to the surgeon, based on the sensed relationshipbetween the surgical plan and surgical execution. A quite sophisticated,"intelligent" system can be used that uses its model of the surgicalplan to automatically customize displays, select appropriate sensortracking modes, and help interpret inputs from the surgeon. In thisultimate system, a helmet-mounted sterographic display could be used toproject the surgical advice directly onto the surgeon's visual field,and the surgeon could use voice input to tell the system what he wants.In a basic system, very simple realtime graphics and auditory cues canbe provided for alignment. For use with bone fragments, an online 3Dmodel display can provide somewhat more detailed "snapshots" of bonefragment positions relative to the surgical plan. The surgeon can have alimited ability to modify the sequence interactively through standardmenus, sterilizable computer input devices, and the pointing system. Forinstance, he could designate where, exactly he has placed (or proposesto place) an osteotomy. The computer could be adapted to simulate thiscutting action "online" and allow the surgeon to compare it with the cutproposed when the surgery was planned.

Passive manipulation aids may be provided to assist the surgeon inprecisely aligning bone fragments or in aligning his instrumentsrelative to the patient. The defining characteristic of such passiveaids is that the surgeon provides all the motive force. Generally, themanipulation aid should interfere as little as possible with thesurgeon's tactile "feel" for what is happening to the patient, whilepreserving the desired alignment once it is achieved. Sixdegree-of-freedom manipulation aids with manually and semi-automaticallyactuated brakes have been developed for tissue retraction, instrumentplacement, and similar applications. One serious drawback of thesesystems is that they provide little assistance in actually achieving thedesired alignment. Even without the additional inertia of a mechanicallinkage, most people find it extremely difficult to achieve an accuratesix degree-of-freedom alignment. The present invention can usemanipulation aids with computer controlled (or manually actuated) brakesto provide selective locking of orthogonally decoupleddegrees-of-freedom resolved in a tool frame centered at a pointreasonably far removed from the mechanism. This permits implementationof a variety of manipulation strategies in which the surgeon only needsto work on aligning a few (often only one) degrees-of-freedom at a time.

Referring now to FIG. 3, there is shown an alternate embodiment of thepresent invention. The apparatus 140 is similar to the apparatus 10 andcomprises a coarse motion manipulator 142 and a fine motion manipulator144. The coarse motion manipulator 142 is substantially the same as thecoarse motion manipulator 12 shown in FIG. 1 and has three links 146,147 and 148. The fine motion manipulator 144 is substantially identicalto the fine motion manipulator 14 shown in FIG. 1. The coarse motionmanipulator 142 shown in FIG. 3, unlike the manipulator 14 of FIG. 1, isintended to be computer controlled.

In the embodiment shown, the first link 146 includes a computercontrolled driver 150 and a computer controlled brake 152. In analternate embodiment only the driver 150 need be provided if it isadapted to also function as a brake mechanism. The driver 150 can be anysuitable type of electromechanical or electromagnetic drive mechanism,such as a rack and pinion screw type drive mechanism where the rails 154have screw threads. The driver 150 is thus adapted to move the mainblock 156 of the first link along the rails 154. The brake 152 may becomprised of any suitable brake mechanism and is generally provided tocontrol the movement of the block 156 on the rails 154. Both the brake152 and driver 150 are connected to the computer 124 which is adapted toactuate the brake 152 and driver 150 to automatically move the firstlink 146. In a preferred embodiment, the surgeon can also manually movethe first link 146 without use of the driver 150 and the brake 152 canbe actuated by the computer to limit or guide movement of the first link146.

The second link 147 also comprises a computer controlled driver 158 anda computer controlled brake 160 attached to its bottom section 162. Thedriver 158 and brake 160, which are both connected to the computer 124,control movement of the second link 147 on the first link 146. In theembodiment shown, the second link driver and brake are substantially thesame as the first link driver and brake. The third link 148 alsocomprises a computer controlled driver 164 and computer controlled brake166. The driver 164 and brake 166, which are both connected to thecomputer 124, control movement of the third link 148 on the second link147. In the embodiment shown, the third link driver and brake aresubstantially the same as the other links' drivers and brakes. However,different types and/or combinations of drivers and brakes could beprovided in the coarse motion manipulator 142.

With the embodiment shown in FIG. 3, the computer 124 can move thecoarse motion manipulator 142 to automatically move the fine motionmanipulator 144 to a location. The surgeon can than use the passive onlyfine motion manipulator 144 to position and move a surgical instrumentconnected thereto. This embodiment allows the computer to move thecoarse motion manipulator 142 with relatively good speed and accuracy,but nonetheless allows the surgeon fully manual fine motion manipulationvia the passive manual only fine motion manipulator 144. Thus, allowingthe surgeon to use his skills during fine motion manipulation. As notedabove, computer control of the coarse motion manipulator 142 canpreferably be manually overriden in the event that the surgeon desiresto manually move the coarse motion manipulator. 142. In addition, in analternate embodiment, the drivers 150, 160 and 164 and/or joints betweenthe links 146, 147, 148 have suitable encoders connected to the computer124 to signal the relative positions of the links relative to eachother. The manipulator apparatus 10, in the embodiment shown in FIG. 2,also comprises a force sensor 95 that is adapted to measure the amountof force being applied to the patient by an instrument connected to theapparatus 10. Any suitable force sensor can be used and, the forcesensor 95 is adapted to signal the computer 124 of the sensed force toprovide force safety monitoring to supplement a surgeon's tactilefeedback. This can be especially important when micrometer adjustmentsare being used.

Referring now to FIG. 4, there is shown an alternate embodiment of adistal end of an apparatus similar to those shown in FIGS. 1-3. In theembodiment shown, the distal end 170 of a coarse motion linearmanipulator third link 172 has a coarse motion rotational manipulator174 connected thereto. The fine motion manipulator 176 is connected tothe coarse motion rotational manipulator 174. The coarse motionrotational manipulator 174 comprises a ball and socket arrangement witha ball member 178 connected to the distal end 170 and a socket member180. However, any suitable type of coarse motion rotational manipulatorcould be provided. The coarse motion rotational manipulator 174 isadapted to provide motion in three revolute degrees-of-freedom as shownby arrows E', F', and G' in FIG. 4. The socket member 180 has frictionarms 182 adapted to be pushed against the ball member 178 to lock thetwo members together and a lock control knob 184 that can be selectivelyactuated by the surgeon to move the arms 182. Alternatively, a computercontrolled driver and/or brake could be provided at the coarse motionrotational manipulator 174 to automatically control rotation, resistanceto movement, and locking. The ball-and-socket joint may be used to alignthe direction of the fine motion axes A', B', and C' in any directionconvenient for the surgeon. If desired, this ball-and-socket joint maybe replaced by a remote-center-of-motion orthogonally decoupled wristmechanism similar to rotational section 70.

The fine motion manipulator 176, in the embodiment shown, generallycomprises a linear motion section 186 and a rotational motion section188. The linear motion section 186 is substantially similar to the finemotion manipulator linear motion section described with reference to theembodiment shown in FIG. 1A. The linear motion section 186 has a firstsection 190, a second section 192, and a third section 194. The firstsection 190 is connected to the coarse motion rotational manipulatorsection 174 and is adapted to provide movement in only a single linearpath of motion or degree-of-freedom as shown by arrow B'. The firstsection 190, in the embodiment shown, also comprises a computercontrolled driver 196 and a computer controlled brake 198 to controlmovement of the first section. The second section 192 also comprises acomputer controlled driver 200 and a computer controlled brake 202 and,is connected to the first section 190 to provide movement in only asingle linear path of motion or degree-of-freedom as shown by arrows A'.The third section 194 is connected to the second section 192 to providemovement in a single linear path of motion or degree-of-freedom as shownby arrow C'. The third section 194 also comprises a computer controlleddriver 204 and a computer controlled brake 206.

The rotational motion section 186, in the embodiment shown, generallycomprises a first rotation section 208, a second rotation section 210, athird rotation section 212 and a fourth rotation section 214. Thus, therotational motion section 186 is adapted to provide movement in fourrevolute degrees-of-freedom. The first, second, and third rotationalsections 208, 210, 212 are substantially similar to the rotationalmanipulator shown in FIG. 1A. However, in the embodiment shown, each ofthese sections also comprise computer controlled drivers 216 and brakes218. The fourth section 214 is adapted to provide rotational movement asshown by arrow I to the end member 94 connected thereto that can movethe tool 110. The rotational axes of paths E, F, G and H, in theembodiment shown, all intersect at a common point.

Each section in the fine motion manipulator 176 is preferably alsomanually movable. Both drivers and brakes need not be provided at eachsection, but may be alternatively provided in different embodiments. Inaddition, manual actuatable locks, such as locks 220 and 221 can beprovided at each section. Micrometer adjustors, such as micrometer 222,may be provided at each section. In a preferred embodiment in whichmicrometer adjustors are provided, the micrometer adjusters are adaptedto provide movement even when a section is locked. The manipulator 176can be used with a passive only coarse motion manipulator, such as shownin FIG. 1, or a computer controlled coarse motion manipulator, such asshown in FIG. 3. One advantage of the manipulator 176 is that it allowsrelatively small rotational alignments of an end effector relative totarget area by only relatively small rotational movements of themanipulator 176 and, relatively small linear alignments of an endeffector relative to target area by only relatively small linearmovements of the manipulator 176. Another significant advantage of thisfour degree-of-freedom design is that it makes it easier to fullyde-couple the rotational motions in a convenient way. In a threedegree-of-freedom wrist, the motions are only really decoupled whenrotational motion F is at its approximate center, so that axes a_(E),a_(F), and a_(G) are all mutually perpendicular. With a four axisdesign, it is possible to align the wrist through a large range oforientations so that axes a_(F), a_(G), and a_(H) are all mutuallyperpendicular, thus providing three orthogonally decoupled motions. Thismakes it possible for the surgeon to select a convenient set of rotationaxes for performing his task. Computer controlled drivers and/or brakescan allow a surgeon to operate with greater speed and/or accuracy.Although beacons 112 are used on the end member 94 to signal a sensorsuch as sensor 122 in FIG. 2, joint encoders can also be provided as aredundant means of sensing the position of the end member 94 in theevent that the direct line of sight between the beacons 112 and sensor122 (see FIG. 2) is blocked.

In addition to determining the position of a surgical instrumentconnected to the apparatus 10 or 140, the optical position sensor systemof the present invention is adapted to use beacons to sense anddetermine the position of a patient, portions of a patient's body, orother surgical instruments and, relative motion of various items.Referring also to FIG. 5, there is shown a beacon 224 and fixation pin225. The fixation pin 225 is adapted to be fixedly connected to aportion of a patient's body, such as bone. In a preferred embodiment,the pin 225 is a "K-wire" known in the surgical art. The beacon 224generally comprises a frame 226, a light emitting diode (LED) 227, andan electrical wire 228 connected to the LED. The frame 226 has a channelsection 229 adapted to receive or be removably positioned on one end ofthe pin 225. The channel section 229 has a center channel 230 with anend stop 231. The LED 227 is coaxially aligned over the center channel230. Because the LED 227 is coaxially aligned over the center channel230, when the frame 226 is connected to the pin 225, the LED 227 iscoaxially aligned with the pin 225. In addition, because the framechannel 230 has an end stop 231, this allows the beacon 224 to beremoved from the pin 225 and replaced again on the pin with the LED 227returning to its same position on the pin as before it was removed. Thiscan obviously give a surgeon greater freedom in an operating room byenabling him to move a beacon and replace it at a later time withoutrisk of causing deviations in position readings by the optical sensor.The center of the visual target will always be located at the same pointrelative to the pin over which it is attached. In a preferredembodiment, suitable retention means, such as a friction sleeve isprovided on the beacon 224 to prevent the beacon 224 from inadvertentlyslipping of of the end of the pin 225. It should be understood thatactive beacons are just one possible sort of visual target. Alternativetargets could include inactive visual targets, such as spheres or disks,so long as the sensing system can accurately determine the target'sthree-dimensional location through a reasonably large viewing angle.

Referring now also to FIG. 6, there is shown a schematic view of apatient's skull S having pins 225 connected thereto and beacons 224mounted to the pins 225. The wires 228 are connected to a computer whichcontrols the timing and/or frequency of illumination of the the beacons'LEDs. In order to determine the position of the beacons 224 relative tothe patient's anatomy, in this case the skull S, a pointer 232 as seenin FIG. 7 is used. The pointer 232 has beacons 233 and a pointer tip234. The surgeon moves the pointer 232 to various landmarks on thepatient's skull S and indicates to the computer where the pointer tip234 is located. The sensor 122 (see FIG. 2) is able to sense theposition of the pointer tip 234 from the positions of the beacons 233and thus, calculates or determines the position of the skull S and thepositions of the beacons 224 on the skull. It should be understood thatany suitable pointer could be used, or any suitable means fordetermining the positions of the beacons on the patient's anatomy couldbe used. Once the positions of the beacons 224 are determined relativeto the patient's anatomy, the computer can calculate or determine therelative position of the surgical instrument connected to the apparatus10 or 140 relative to the patient's body by sensing the positions of thebeacons 112 relative to beacons 224.

In addition to sensing the relative positions of a surgical instrumentrelative to a position of a patient's anatomy, the present invention canbe used for positioning a first portion of a patient's body relative toa second portion of a patient's body. This can be particularly usefullyin procedures such as reconstructive plastic surgery of a patient'sfacial skull area. For example, if the bone fragment or piece I wasintended to be repositioned relative to the remainder if the skull S,the sensing system of the present invention could sense the positions ofthe beacons 224 relative to each other. In a preferred embodiment, thecomputer would already have a surgical plan and optimum position of thebone fragment I. Once the beacon connected to the bone fragment Ireaches a predetermined position, the computer would signal the surgeonof the accomplishment. Furthermore, the decoupled degrees of freedom ofthe manipulation aid permit each degree-of-freedom to be alignedseparately and successively, which is much easier than trying to alignall degrees-of-freedom simultaneously. Obviously, computer sensing anddetermining of relative positioning is highly more accurate than merelya surgeon's visual sighting. In a preferred embodiment of the system,the apparatus 10 (see FIG. 1) has a bone clamp adapted to clamp onto thebone fragment I and move the fragment I with computer controlled brakesat each joint of the apparatus 10 to assist the surgeon in fast andaccurate relative positioning of the fragment I. Of course, the presentinvention can be used in any suitable surgical operation.

Referring now also to FIG. 8, there is shown a schematic view of twosurgical instruments 236 and 237 passing through incisions 238 and 239into a patient's body P and, more specifically, is intended toschematically show a laparoscopic operation. The proximal ends (notshown) of the instruments 236 and 237 are connected to two fine motionrotational manipulators, such as shown in FIG. 4, that can independentlymove the instruments. Because of the unique motion structure of the finemotion rotational manipulators, the center of rotation for eachinstrument is located at the incisions 238 and 239 remote from themanipulators. Obviously, in addition to laparoscopy, the system may bereadily adapted to arthrosopic or similar surgical procedures in which asurgical instrument must be placed into the patient's body through asincision, body orifice, or other restricted opening, so that little (ifany) lateral motion is possible at the point of insertion. Typically inthe past, the surgeon's manipulation of the end of the instrumentoutside the patient's body in order to place the end of instrumentinside the body in a desired relationship to the patient's anatomyand/or to other surgical instruments was rather cumbersome and, precisemanipulation was very difficult. Often, two people .were needed tomanually, manipulate instruments (e.g., one person to hold the camera orcameras and one person using both hands to operate two cutting ormanipulation instruments).

Referring also to FIG. 9, there is shown a schematic view of a finemotion rotational manipulator 240 with an instrument 241 insertedthrough an incision into a patient. In the embodiment shown, themanipulator 240 is adapted to provide axial sliding motion to theinstrument 241 to provide a four degree-of-freedom remotecenter-of-motion wrist, which is supported by a coarse positioningsystem with three orthogonal linear degrees of freedom. The coarsepositioning degrees of freedom are use to place the center-of-motion Mof the remote center-of-motion wrist at the point of insertion into thepatient's body P. The three most distal revolute degrees-of freedom andthe sliding degree of freedom give the surgeon an orthogonally decoupledfour degree-of-freedom spherical work volume centered at the insertionpoint. These degrees-of-freedom may be selectively locked or movedindependently to assist a surgeon in achieving a desired precisealignment. Furthermore, small motions within the four degree-of-freedomwork space can be achieved with only small motions of the individualaxes. A proximal revolute degree of freedom ⊖_(p) whose motion axespasses throught point M may be used to orient the distal remotecenter-of-motion axes for maximum convenience. This degree of freedom⊖_(p) may be moved during the procedure without disturbing the alignmentof the manipulator with the point of insertion.

Referring also to FIG. 10, there is shown a schematic view of a systemfor use in laparoscopy or similar procedures. The system generallycomprises a manipulator apparatus 242, a computer 243, a drive motorinterface 244, a monitor 247 with a suitable image processor 245 andgraphics adaptor 246, and a terminal 248 for connecting additional inputdevices to the computer 243. The manipulator 242 has drive motors 249 atits three rotational sections 250, 251, and 252 to provide ⊖_(x), ⊖_(y)and distal ⊖_(z) motion and, a slide motor 253 adapted to axially slidethe instrument 254. The manipulator 242, in the embodiment shown, alsohas proximal end X, Y, Z and ⊖_(z) motion capability that can be locked,such as by lock 255. In the embodiment shown, the three most distalrevolute axes and the sliding axis are equipped with locks and motorizedmicrometer fine adjustments. The motors 249 and 253 in turn, are driventhrough the appropriate interface 244 by computer 243. The instrument254, in the embodiment shown, includes a splitter 256, a filter 257, abeacon detector 258, a video camera 259, and a suitable image bundle(not shown) to transmit an image from the distal end of the instrument254 to the splitter 256. The splitter 256 divides the image into twoimages allowing one image to be received at the camera 259 and, theother image passed through the filter 257 to be received at the beacondetector 258. The video output of the camera 259 is fed into thegraphics adaptor 246, where it may be optionally mixed with graphicsoutput from the computer, and is then displayed to the surgeon onmonitor 247. The video output of the camera is also optionally fed intoimage processing system 245 which analyzes the images produced by thecamera and provides information to the computer about the relativeposition of the surgeon's instruments, the camera, and the patient'sanatomy. In the embodiment shown, the surgeon is using a secondinstrument 260 inside the patient's body which has beacons 261 connectedthereto. The beacons 261, in the embodiment shown, are comprised ofdistal ends of fiber optic bundles 265 connected to an illuminationsystem 262 located outside the patient. The illumination system 262includes a flasher 263 controlled by the computer 243 and LEDs 264.Light is transmitted by the fiber optic bundles 261 to their distal endson he second instrument 260 where it can be viewed by the firstinstrument 254.

In the preferred embodiment, the fiber-optic beacons are used to providereadily distinguished targets to facilitate tracking of the surgicalinstruments. In this case, the LED light sources are bonded to theoptical fibers. The distal end of each fiber is bonded in a known placeto the surgical instrument to be tracked, and the end is shaped toprovide a point source of light visible through a wide angle. Multiplefibers may be bonded to the instrument and may be flashed in successionunder control of the computer. The optical splitter 256 diverts aportion of the light passing back through the camera's optical system.This light is passed through a narrow band-pass filter 257 matched tothe wavelength of the LED beacons and is imaged onto an appropriatedetector 258, which determines the position in the image plane of thecentroid of the image of each beacon. From this information, thecomputer can determine the direction of each point light source to thetip 266 of the camera optical system. From several such point sources,the computer can compute the location and orientation of the surgeon'sinstrument in the patient's body relative to the tip 266 of the cameraoptics. Alternatively, the same information may be obtained throughstandard image processing techniques from the video camera.

The computer can compute appropriate motor commands to move the camerato achieve a desired relationship with the surgeon's instrument. Thesystem can include a method of aligning the manipulator center of motionto a body part and then grabbing the part with an adjustable clamp. Inthe simplest case, the computer could simply command appropriate ⊖_(x)and ⊖_(y) commands to place the image of a single beacon attached to aninstrument at any desired place (e.g. the center) in the video imageseen by the video camera. This could be done either in a continuouslytracking mode or on command from the surgeon. In more complex cases, thecomputer would also use the distal ⊖_(z) and sliding motors to provide a"roll" and "zoom" capability.

The surgeon can also use this instrument tracking capability todesignate particular anatomical features to the computer or to designatemodifications in his surgical plan. The computer can then use thisinformation to modify the graphics display provided to the surgeon, tomodify the parameters or algorithms selected for image processing, or incontrol of additional manipulation aids or system used in surgery. Forexample, the surgeon may indicate a few points on a blood vessel, bileduct, or other anatomical feature that he wishes to inspect. Thecomputer can then scan the camera over the indicated path while thesurgeon devotes his whole attention to the display. Furthermore, thecomputer can use the designation information to customize imageprocessing algorithms to enhance what the surgeon sees. Clearly, thesystem described above can be modified for use with surgical instrumentsother than cameras. For example, a pair of remote-center-of-motionmanipulation aids may be used to assist laparoscopic alignment of twoanatomical features (e.g., the several ends of a blood vessel or duct)that must be precisely aligned for joining. In one system of the presentinvention in addition to visually displaying the path of a surgicalinstrument, the computer can determine a path from a sensed position ofthe surgical instrument to a target position. The computer can alsomonitor movement of the surgical instrument and audibly signal deviationfrom the path. Thus, a surgeon does not need to continuously observe avisual comparison of the determined path and actual path. The audiosignals can be provided as different types of signals for differenttypes' of occurrences with the computer selecting the appropriate signalbased upon a sensed occurrence. The sensing system may also includedifferent sensing modes, such as coarse and fine, and the computer maybe adapted to select different sensing modes based upon the sensedposition of a surgical instrument relative to a surgical plan, a targetarea, and/or another surgical instrument, etc. The different sensingmodes might also be things such as yaw, pitch, and roll position of thesurgical instrument. Thus, the computer could also be adapted toautomatically select different types of advice to give the surgeon basedupon the surgical plan and the sensed surgical execution. Another systemcould include means for inputting a change in the surgical plan into thecomputer during surgery and determining a new surgical plan based, atleast partially, upon previously sensed surgical execution of thesurgical plan. The means for inputting could include the surgeon movinga surgical instrument to a new position and the means for sensing beingadapted to sense the new position and communicate the new position tothe means for determining. The means for inputting could be a voiceresponsive input system. The means for advising could include activeadvice such as automatically providing resistance force to motion of asurgical instrument in at least one degree-of-freedom, or, automaticallymoving the surgical instrument.

Let it be understood that the foregoing description is only illustrativeof the invention. Various alternatives and modifications can be devisedby those skilled in the art without departing from the spirit of theinvention. Accordingly, the present invention is intended to embrace allsuch alternatives, modifications and variances which fall within thescope of the appended claims.

What is claimed is:
 1. A method of tracking position of a surgicalinstrument inside a patient's body comprising steps of:inserting a firstsurgical instrument into a patient's body, the first instrument having avisual target thereon; inserting a separate second surgical instrumentinto the patient's body, the second instrument having means fortransmitting an image out of the patient's body; and moving at least oneof the surgical instruments automatically based upon movement of theother surgical instrument to substantially continuously transmit animage of the visual target by the means for transmitting.
 2. A method asin claim 1 wherein the step of moving comprises keeping the visualtarget at a predetermined position in the image.
 3. A method as in claim1 wherein the visual target is an energy emitting beacon and the step ofmoving comprises a computer connected to the second instrument and, themeans for transmitting automatically moves the second instrument basedupon energy from the beacon being viewed by the means for transmitting.4. A method of performing surgery inside a patient's body, the methodcomprising steps of:inserting a first surgical instrument into apatient's body, the first instrument having a visual target thereon;inserting a second surgical instrument into a patient's body, the secondinstrument having means for transmitting an image out of the patient'sbody; placing the first instrument at a selected position relative tothe patient's anatomy; observing, by a controller connected to the meansfor transmitting, an image of the visual target that is transmitted bythe means for transmitting; and moving, by the controller, at least oneof the surgical instruments automatically based upon an observedposition of the visual target.
 5. A method as in claim 4 wherein thevisual target includes at least one beacon that emits energy, the stepof observing having the movement controller observe the energy emittedby the beacon.
 6. A method as in claim 4 wherein the step of movingautomatically moves the surgical instrument over a path of previouslyobserved positions.
 7. A method as in claim 4 further comprisingdetermining a position of a selected anatomical feature of the patient'sanatomy by the controller based upon, at least partially, the image ofthe visual target observed by the controller.
 8. A method as in claim 7further comprising the controller modifying, through image processing,images transmitted by the means for transmitting based upon the positionof the selected anatomical feature.
 9. A method as in claim 7 furthercomprising the controller automatically determining a path for the firstsurgical instrument to travel from a present position of the firstsurgical instrument to the selected anatomical feature.
 10. A method ofsurgery comprising steps of:inserting a first surgical instrument into apatient's body, the first instrument having a target thereon; insertinga separate second surgical instrument into the patient's body, thesecond instrument having means for transmitting an image out of thepatient's body; transmitting an image from inside the patient's body bythe transmitting means to a controller, the image including the target;and automatically moving, by the controller, the second surgicalinstrument when the target is moved to maintain position of the targetat a desired location in the image transmitted by the transmittingmeans.
 11. A method as in claim 10 wherein the target includes at leastone beacon that emits energy, the step of automatically movingcomprising the controller monitoring the position of the energy emittedfrom the beacon in the image transmitted by the transmitting means.